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ABSTRACT 
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the United Kingdom education research community. Using insights from 25 
interviews with key stakeholders, it seeks to describe what the current 
strengths and weaknesses in methods are, and the methodological developments 
that are needed for the future health of the field. Using survey returns from 
521 active researchers, the paper goes on to describe the techniques that the 
field can use, and those where further "training" or experience is required. 
Using the 8,691 individual Research Assessment Exercise returns (from a 
survey of higher education research activities) , the paper then summarizes 
the methods reported to be in actual use. Finally, it uses a brief analysis 
of journal contents as triangulating evidence. The informants were generally 
in agreement that there is currently a widespread weakness in the quality of 
UK education research. Much of this weakness is attributed by them to a 
shortage of skills in quantitative methods. Other data sources suggest that 
the latter is less likely than the informants believe. A clear majority of 
education researchers report having used quantitative methods, and the 
substantial number of publications involving quantitative methods supports 
this view. It is, perhaps, rather the type and quality of both quantitative 
and qualitative research that leads to the poor public image of education 
research. Improvement is not going to come simply by enlarging the group of 
people using quantitative methods. Two appendixes provide information about 
the data collected. (Contains 23 references.) (SLD) 
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Abstract 

This paper considers the range of research methods used by the UK 
education research community. Using insights from 25 interviews with 
key stakeholders it seeks to describe what the current strengths and 
weaknesses in methods are, and what methodological developments are 
needed for the future health of the field. Using survey returns from 521 
active researchers, the paper goes on to describe the techniques that the 
field can use, and those where further ‘training’ or experience is required. 
Using the 8,691 individual RAE returns to education, the paper then 
summarises the methods reported to be in actual use. Finally, it uses a 
brief analysis of journal contents as triangulating evidence. 

Our informants were generally in agreement that there is currently a 
widespread weakness in the quality of UK education research. Much of 
this weakness is attributed by them to a shortage of skills in ‘quantitative’ 
methods. Our other data sources suggest that the latter is less likely than 
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the informants believe. A clear majority of education researchers report 
having used quantitative methods, and the substantial number of 
publications involving quantitative methods supports this view. It is, 
perhaps, rather the type and quality of both quantitative and qualitative 
research that leads to the poor public image of education research. 
Improvement is not going to come simply by enlarging the group of 
people using quantitative methods. 



Introduction 

This paper uses data derived from interviews with stakeholders, a survey 
of education researchers, an analysis of the methods reported in the 200 1 
RAE returns, and a summary of the methods used in a small number of 
strategically-selected Journals during 2002. Its purpose is to describe the 
methods currently in use in education research, and consider the ways in 
which the capacity to do relevant high-quality research might be 
enhanced. 

The funding mechanisms for UK Higher Education Institutions have 
highlighted the importance of comparative measures of research 
excellence. One consequence of this has been the Research Assessment 
Exercise (RAE), carried out in 1992 by the then Universities Funding 
Council to determine the quality of research undertaken within the higher 
education sector. The RAE, repeated in 1996 and again in 2001, has 
provided extensive information for comparing research activity across the 
whole of the UK. In so far as the criteria can be said to be the same over 
time, the results of the 2001 RAE record an overall improvement in 
research quality through universities and colleges across the UK 
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(HEFCE, 2001). 55% of all research active staff now work in 
departments graded 5 or 5* compared to 31% in 1996. In addition to this, 
research at the lower end of the scale (rated 1 or 2) previously accounting 
for a quarter of all research has now fallen to just 6 per cent (HEFCE, 
2001). In 1996 43% of all HEI submissions were graded with 4, 5 or 5*, 
this rose to 64% in 2001. Some of this improvement is due to the 
substantial number of persistently low ranking HEIs choosing not to enter 
for the 2001 RAE due to little or no chance of funding allocation (The 
United Kingdom Parliament 2002). For 2001, of the 174 institutions that 
submitted 75% of the funding was allocated to just 24 institutions 
(McNay, 2003). 

The situation for Education (UoA 68) however was somewhat different. 
In 2001, 29% of submitted researchers were in 5/5* departments 
(compared to 55% across all subjects), and 17% of submissions were 
rated 5/5* (compared to 39% across all subjects). There has not been 
anything like the equivalent growth for this subject. There may, of course, 
have been mis-classified submissions (in either direction) and significant 
education-relevant returns to other panels, but the scale of the difference 
between education and others subjects is remarkable. Surely the simplest 
explanation overall is that the results for education suggest a relatively 
poor quality of research, or at least a pressing need to improve the way in 
which it is reported. McNay (2003) points out disparities in the exercise 
concerning funding allocation, consistency of sampling and quality 
assessment definitions for each UoA. Therefore, it is possible to argue 
that Education has been unfairly treated by its panel, and that the quality 
of submitted work is actually better than these grades show. However, 
this is a double-edged argument since it also allows that the panel could 
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have been overly generous to their own discipline, and that the quality of 
education research is even worse than recorded. 

When these results are put together with a series of widely-publicised 
criticisms of UK education research, then it seems there may be a case to 
answer. In recent years there has been mounting attention of the quality 
of education research both within the research community and in a wider 
public policy sphere (Richardson 2002). This can be partly attributed to a 
number of high-profile publications (Hargreaves, 1997, 1999; Hillage et 
al, 1998; Woodhead 1998, and Tooley and Darby 1998) which have 
questioned standards of education research activity currently being 
undertaken within HEIs nationwide. Of these reports, only Tooley and 
Darby’s commentary was based on an, albeit limited, study of empirical 
evidence (264 articles were scrutinised in four journals to ascertain and 
reflect standards of good practice in educational research), and somewhat 
surprisingly generated the greatest opposition to its findings (no evidence 
apparently being preferred to meagre evidence). To what extent are the 
above criticisms justified? 

One of the initial objectives of the ESRC Teaching and Learning 
Research Programme (TLRP) Research Capacity Building Network was 
to undertake an extensive consultation exercise in order to identify the 
priorities for research capacity-building and to generate a database of 
expertise from across the UK educational research community. This 
exercise has involved four elements. The first was to interview key 
stakeholders from across the education field, from policy-makers and 
practitioners to funders and researchers. The second part of the 
consultation exercise has been to survey, as widely as possible, the 
educational research community in order to identify current expertise in 
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research and future training needs. The third element was to undertake a 
simple review of the ‘best’ education research literature, as determined by 
publications returned to the 2002 Research Assessment Exercise (RAE). 
The final element to our consultation exercise involves an audit of the 
methods used in papers published in a small number of journals chosen 
on the basis of the third stage. 

This paper reports some of the findings from these four elements. It 
focuses on the range of methods currently being used by academic UK 
education researchers. The range of data sources used here means that 
there is no room to discuss wider issues such as the relevance and impact 
of research, and there is no specific mention of the role of practitioner 
research. Nor do we reach definite conclusions about what should be 
done. The paper deals with each element in order, briefly describing our 
own methods of data collection and analysis, then summarising the 
results of each. 



What are the beliefs about current capacity? 

Stakeholder interviews 

We interviewed twenty-five key stakeholders, each representing the 
major constituencies of the UK education community, in 2001/2002. The 
stakeholders were chosen to represent the major constituencies of the 
education research world. They included representatives from the 
following: 

• national and local government - comprising elected members, policy- 
makers, and researchers; 
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• research funding agencies; education research organisations - such as 
the British Educational Research Association (BERA), the National 
Foundation for Educational Research (NFER), the National 
Educational Research Forum (NERF), the Office for Standards in 
Education (OFSTED) and the ESRC Teaching and Learning Research 
Programme (TLRP); 

• HE education researchers - including TLRP project leaders, non- 
TLRP researchers, and Research Assessment Exercise (RAE) panel 
members; 

• the teaching profession — including the General Teaching Council and 
the Teacher Training Agency; 

• education research journal editors; 

• and stakeholders with an international perspective (OECD). 

The semi-structured interviews concerned the nature of research capacity, 
the current capacity of the UK community, the role of the respondents’ 
own body, and ideas for capacity building. In particular, they were asked 
about the current state of education research in the UK, why it is like this, 
and how education research could continue to move forward. Interviews 
were recorded, and transcribed, and then sorted according to a variety of 
themes. Here the emphasis is on the methods used in research, and the 
need to improve the quality of research. 



Selected findings 
Quality 

When asked about the current state of education research many of the 
stakeholders began, without prompting, by addressing the published 
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criticisms referred to above. These have clearly had a major impact upon 
the stakeholders and prompted them, if they had not already done so, to 
reflect upon the state of education research in relation to their own 
experiences. Notably, while written public reaction to these commentaries 
has tended to be defensive about the current state of education research 
(e.g. Hodkinson 1998), the response of all stakeholders was to express 
some agreement with the criticisms. Of course, several also stressed that 
some research was, and always had been, extremely good. 

The best research was always very good and the best research is still 
very good. (HE researcher and RAE panel member). 

There is some very good research done and some very poor research 
done. (Education researcher and BERA Executive Council member) 

I mean I have had for some time a concern about educational 
research that because it comes from such a variety of institutions and 
individuals that it probably has greater variation in quality than 
research in almost every other area. (HE researcher and RAE panel 
member) 

Even when the stakeholders expressed concern about the motivation for 
the public criticisms, these are seen as ultimately beneficial. 

I think a lot of the current concerns around capacity building go back 
to those criticism and people say "well ok we've got to strengthen 
educational research, we've got to make it more relevant and so on". 
And maybe some good has come out of it. (HE researcher and TLRP 
Team Leader) 
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The consensus was that the criticisms had some basis in fact. 



I think well certainly I and other people I know actually go along 
with a great deal of what David Hargreaves says (HE researcher and 
UCET Executive Committee member) 

I am starting to sound like David Hargreaves. He had a point 
actually. The butt of jokes in education departments, who think he is 
terribly crass to say this, but I think some of the stuff that you look at 
is rubbish. (Education researcher and BERA Executive Council 
member) 

It always seems to me with David Hargreaves that he is extremely 
good at getting things on the agenda, in order to do that he always 
makes a case, nevertheless, there is always something in what he 
says. (HE researcher and RAE panel member) 

There is a lot of stuff talking about policy, but it wasn't helping 
anyone. When you bring it down to the technical level very large 
amounts of it was technically so bad as to be embarrassing. No 
discussion about validity and reliability of data for example. No 
attempt to theorise. (HE researcher and member of NERF) 

I mean I to be honest have been surprised sometimes with what I 
see as the relatively low level of sophistication in some of the 
research that’s done. (OFSTED research manager) 
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Lots of low level cottage industry stuff. Too little understanding that 
it’s actually a skilled, time consuming business not to be engaged in 
lightly or amateurishly. That sounds a bit hard but as somebody that 
edits two journals and sits on four or five editorial boards I see a lot 
of the stuff which people see as their best shot. If you’re submitting 
something for a journal you’ve really worked it over. Some of the 
stuff that I see.... thank Christ somebody has brought this for 
somebody else to look at never mind considered to be published. 
(HE researcher) 

The people who are doing the current EPPI reviews, the review 
groups, keep coming back and saying how shocked they are by the 
poor quality of the literature. (DIES Officer) 

I think the quality of the policy-oriented research has been poor 
(OECD researcher) 

Well strangely enough I had some sympathy for some of the things 
Chris Woodhead said about it, because a lot of educational research 
frankly doesn't tell you a lot that you don't already know... 
sometimes you question whether in fact it’s been done very 
thoroughly... I think the money we spend is not spent wisely at all. I 
do share the criticism. (Chair of a LEA Education Committee and 
Local Government Association executive member) 

So there is considerable agreement (privately at least) that there is 
something amiss in the standard of UK education research. We, therefore, 
cannot agree with Hodkinson (1998) and others who seek to defend 
themselves from this criticism by asserting that there is no problem of 
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quality, and so no need for us to improve our research. But neither does 
this lead us to the conclusion attributed to Chris Woodhead that education 
research is so poor that it should no longer be funded. We need to 
improve. So what exactly could we improve in our capacity to undertake 
research? 

Problems of method 

Again, there was near consensus among the respondents. While other 
issues, such as impact, were emphasised the core issue about quality 
concerned the lack of skills in key methods. A number of stakeholders 
referred to the lack of research ‘skills’ among the research community. In 
this example the deficit referred to concerned knowledge of ‘quantitative’ 
techniques. 

Actually the situation is so bad that people don ’t realise they haven’t 
got the skills they need which is even worse than having people who 
are struggling but aren’t sure where to get the skills they need. 
(DIES officer) 

There aren’t the people in the system to provide the research that we 
want, we know that. (DIES officer). 

I mean, I think the quality of some people in education is highly 
questionable. I can point to you, I could point the people walking 
around with PhDs, who demean the title, right. (HE researcher). 

What I think the weakness of a lot of the proposals were and the 
reasons they weren't funded was the kind of lack of theoretical 
rigour and methodological rigour and I think that’s where 
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educational research is weak. (HE researcher and TLRP Team 
Leader). 

This last respondent is referring to their experiences as a Board member 
for the ESRC. Another common criticism of research is that the 
conclusions made bear no relationship to the evidence presented (i.e. they 
are not warranted, a point we return to in the concluding section). In 
particular, the stakeholders believed that research over-generalised, 
largely because the majority of education research is considered small- 
scale and non-cumulative. 

There is a gulf between your evidential base and your conclusions, 
so sometimes it is like that, quality issues, but other times you 
caimot understand the relevance of the whole area of study. I would 
just say it is always a problem about generalising from a small 
sample. (Education researcher and BERA Executive Council 
member) 

It is at this point that many stakeholders start to differentiate between 
qualitative and quantitative research methods. There was a clear 
perception that there is a lack of quantitative research but, as we shall see, 
the issue of quality and rigour was raised equally for quantitative and 
qualitative research. 

Too little Quantitative? 

The purported shortage of ‘quantitative’ research in education, and the 
social sciences more generally, has been given a very high profile in 
recent years (c.g. Marshall 2001). The issues of causality, being able to 
test propositions, and the process of generalisation raised by many 
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stakeholders are more complex than simply the problem of a lack of 
quantitative research. But nearly every stakeholder reported that there 
was a regrettable lack of quantitative research, and that this has 
straightforward consequences for the quality and relevance of research. 

There is a widely acknowledged absence of quantitative research of 
particular kinds, especially, there's a weakness, there's a relative 
absence and there's no mechanism for addressing that currently. (HE 
researcher and TLRP Team Leader) 

One consequence of this shortage is that there are simply too few 
researchers to pass such skills on to their peers or the next generation of 
researchers. 

It’s a real worry in this department that you know in a department of 
3,500 students and 190 staff there isn’t one strong quantitative 
lecturer. (HE researcher and UCET Executive Committee member) 

I do feel that we are short of good quantitative researchers in this 
country and that is a particular need. (HE researcher and TLRP 
Team Leader) 

Where I think that has failed in a way is when it has not kept the 
proper balance, as there are plenty of questions that have to be 
addressed through quantitative methods. (HE researcher and RAE 
panel member) 

As a journal editor what worries me slightly is that you have so 
much qualitative stuff It’s very very unusual to get anything 
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quantitative. (HE researcher and UCET Executive Committee 
member). 

Poor quality qualitative 

So there is reasonable consensus among our stakeholders that education 
research needs improving, and that the greatest weakness is the lack of 
quantitative work. There is, clearly, a considerable amount of qualitative 
work and this could be construed a strength. However, there are also 
concerns over the quality and appropriateness of this work. There was 
agreement amongst stakeholders that ‘qualitative’ research, although in a 
healthier state than ‘quantitative’ research, has its own limitations. 

Actually there is a quality issue with qualitative research. (HE 
Researcher and TLRP Steering Committee member). 

On the other side there's an awful lot of qualitative workers where 
the quality is not very high, as it were, for various reasons. (Chief 
Executive of research funding agency) 

In qualitative research because certainly when you start to read 
some of this, and clearly some of it is very good, and some of people 
working it, have led the field themselves. I mean, that’s, you know, 
that you can take that group out but below that there seemed to be 
quite a lot of people who are doing qualitative research which isn’t 
terribly good and that maybe because they haven’t ever been very 
well trained in it themselves. (HE researcher) 
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Perhaps, as the next respondent suggests, we are in danger of mutually 
condoning the poverty of our research skills (a point about deficiencies in 
the peer-review process). 

There’s beginning to be a pattern with qualitative papers. I’ve 
noticed it not only in my own journal, but in my case. . . I don’t allow 
them to come through the journal, but in other journals the stuff that 
I read... the pattern is a literature search, then without any real 
critical analysis straight into what the dataset... how it was 
generated and then a quick discussion and that’s it. And that's a kind 
of paper by numbers and you see more and more of them. Now and 
of course no-one then goes back to them and starts critiquing them 
because they are too busy doing the same thing themselves. (HE 
researcher and UCET Executive Committee member). 

So I would say the typical qualitative methodology for a project in 
educational research is interviews and there isn’t much else there 
[...] I suppose, that in a sense, you wonder whether they’re using 
that method because they think it’s the best method, which it may be 
for some things or whether they don’t know any other methods or 
they don’t feel comfortable with or experienced enough with other 
methods to try using those... some people using focus groups but 
again you don’t always get the sense that they actually understand 
very much about what they are, only that they’re, you know, they’re 
not single interview, they’re a group interview. (HE researcher). 

Almost everybody does semi-structured interviews and then mucks 
about for better or worse in a rather unsystematic analysis of what is 
going on so in the end, when I read something and I have no idea at 



the end whether the findings really related to what the interviewee 
thought or simply how they have responded to the researcher’s 
agenda. (HE researcher and RAE panel member). 

The overriding concern was that the use of interviews was the primary 
tool in most research, and that this was often chosen as a method because 
researchers did not have experience in, or training in, other skills - both 
qualitative and quantitative. There was also concern that many 
researchers employed qualitative methods (and perhaps quantitative 
methods) without rigour and without a thorough concern for the way such 
data had been analysed or used. 

As this last respondent shows, a key concern was about the extent to 
which current methods approaches are actually appropriate to their 
purpose. Perhaps some of the poor quality qualitative work should 
actually have had a strong quantitative component. 

There are whole areas of social science where for various reasons 
people have got into the mode disproportionately of doing non- 
cumulative small-scale usually flawed pieces of quote unquote 
qualitative work. They'd have an interesting problem, perfectly 
interesting idea, really plausible you know good ideas about it and 
they thought they could test the ideas by just chatting to 20 kids or 
kids in this classroom versus that classroom and they'd never heard 
of Hawthorne effects or they'd never thought about intentionality 
and social mechanisms or whatever. I'm slightly exaggerating for the 
sake of the point. I mean I think that there are too many examples 
and education is not the only one where in social science people 
have got an interesting problem, some interesting ideas of what the 

O 

ERIC 




16 



answers might be, but they can't do the research necessary to quote 
unquote test their propositions because they don't have the skills. So, 
what they do is they do the research that they can do to match the 
skills that they've got. So, they body swerve around the direct way of 
addressing the question and they have some second best or third best 
take on it that they get from using the only technique or methods that 
they can use and that’s a characteristic weakness of a lot of social 
science and often because people have got this hang up about 
quantitative work disproportionately that’s the bit they avoid. (Chief 
Executive of UK research funding agency). 

Poor quality Quantitative 

Interestingly, even the apparently limited amount of quantitative work 

received quite general criticism of its quality (see also examples in 

Gorard 2003). 

I think you can get terrible quantitative work, there are people who 
just think there's a kind of non problematic general linear model 
reality out there and you just tag variables and start with race and 
that’s it. . . We don't want a generation of people who are trained, 
kind of cloned, trained monocular vision, kind of stimulus response, 
there are three variables, must be a log linear analysis, switch on 
SPSS, press the buttons with all the defaults on is garbage in garbage 
out. (Chief Executive of research funding body). 

Yes, there is a lot of qualitative work and there is less quantitative 
work which is in any way sophisticated, that is, there are lots of 
people doing surveys but when you actually read the outputs from 
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those, they are really not much more than people being able to push 
some buttons on SPSS. (HE researcher). 

You get people who don't think about causality, who think variables 
do things and who just do this ‘hey, path analysis is the technique at 
the moment’. And they never stop to think wait a minute we have to 
talk to some people to answer this question or whatever. (Chief 
Executive of UK research funding agency). 

A number of stakeholders highlighted positive recent developments such 
as the increasing use of multi-level modelling and the analysis of 
complex datasets. But there was a fear that even though such 
developments were being made there were too few researchers who could 
actually use these techniques (perhaps leading to intellectual cronyism). 
There was also the concern that the requirements of these sophisticated 
techniques meant that they had become overly technical and far removed 
from most researchers’ minds. 

I have a suspicion, more than a suspicion, I think there are quite a lot 
of signs that we are losing the quantitative side, just at the point 
when we’re getting more sophisticated techniques, which I think 
we’ve got to be very, very vigilant because, there is a sort of 
technicist mentality that gets seduced by... essentially arithmetic or 
numbers, you know, and, yeah, you’ve got to be very careful about 
that. (HE researcher). 

This would suggest that the situation is not just one of a shortage of 
quantitative research, but also a shortage of the quantitative research 
skills required to be able to understand, and critically review, quantitative 
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research. The following stakeholder suggests that in LEAs where there 
are the people to undertake quantitative data analysis they do not have the 
research background to review the methods they are employing, 

What I am saying is that although there are a lot of people doing 
quantitative work, with school data and pupil data, a lot of them are 
actually implementing central Government analysis. They are not 
necessarily people who have an analytical background in research 
and quantitative research. They are not necessarily being as critical 
about the methodologies and are not as aware of the limitations of 
the methodologies as maybe they ought to be, so you know, there is 
a bit of an uncritical implementation of government strategy, but the 
government is not worried about that. (Former LEA researcher) 

As discussed above, one of the most significant ‘deficits’ in education 
research is in the lack of quantitative methods. It is not surprising, 
therefore, that nearly every stakeholder addressed the issue of building 
capacity in quantitative education research. Not only would this help to 
extend the range and balance of methods in education research but some 
stakeholders could see how this would help develop better quality 
research irrespective of the methods used. 

There is a real problem about the shortage of people who can do 
quantitative work and you can use that to get people to think in a 
more rigorous way about the logic of social research generally. 
Because quantitative researchers have to think formally and much 
harder about things like measurement, indicators, reliability, 
sampling, all the things that all researchers actually ought to think 
about, and Bob Burgess, who started this thing in the training, with 
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the training board, was trying to get, but he couldn't figure out how 
to do it, because when people do non-quantitative research they tend 
not to think nearly as rigorous about the kind of problems that you 
talk about in the paper [see Gorard 2001] and about problems of 
generalisability and learning experimenter effects and the nature of 
intention and blah blah blah. (Chief Executive of a research funding 
body). 

Many stakeholders believed that building capacity in the use of 
quantitative methods is not just about increasing the volume of such 
research, but should focus, in particular, on the more basic level of 
‘quantitative’ techniques, (such as those recommended by Gorard 2003). 

I think I agree with that general thrust that we need more 
quantitative researchers, but I'm not sure it’s the very, very, 
sophisticated techniques we're talking about. It’s more something 
that’s a bit more accessible and usable. (HE researcher and TLRP 
Team Leader). 

This also requires a firm grounding in the assumptions and principles 
behind many ‘quantitative’ methods. This, as many stakeholders are keen 
to point out, may be necessary for those who already undertake 
quantitative research but have been trained in ‘button-pressing’. 

Why is it like this? 

In this final section based on the interviews, we consider possible reasons 
for the situation above, and thus possible avenues for improvement. Here 
we focus on methods and in particular the quantitative/qualitative 
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‘divide’. Our respondents clearly also commented on a much wider range 
of issues with appropriate plans to assist improvement (see Taylor 2002). 

It may appear safer to do qualitative research because it is seen as being 
more subjective and hence difficult to be criticised, meaning that weak 
work can survive scrutiny by peers (particularly if this standard is 
mutual). 

This maybe a bit crude, but its easier to show that something is 
wrong statistically than through the touchy-feely stuff, so your 
reputation is less amiss if you’re doing the qualitative, I think. (HE 
researcher and UCET Executive Committee member). 

Perhaps this is partly why the Tooley and Darby (1998) report, which 
focussed on qualitative pieces, was such a shock to those involved. 
Another stakeholder argued that qualitative research is presented as being 
easier to do than quantitative research, thereby encouraging new 
researchers to undertake qualitative research, and giving a false 
impression of qualitative research as being not needing to be as rigorous 
or scientific. 

I think sometimes qualitative research is presented as being easier 
than quantitative research. That the concepts in quantitative research 
require you to be numerate, they require you to think quantitatively 
and that to be a qualitative researcher all you need to do is go out 
with a tape recorder and get some quotes. Now we know it isn't as 
easy as that. (HE researcher and TLRP Team Leader) 



Another common response was that the choice of research method was 
related to the creation of ‘methodological identities’. These identities, 
some stakeholders argued, were generated very early on in the 
development of researchers’ knowledge and skills. They can severely 
restrict our approach to research training, and reinforce the selective 
ignorance of work involving numbers. 

It’s just that they say at an early stage, they say “right I'm going to 
do an ethnography or I'm going to use qualitative interviews, I really 
don't want to know how to do anything quantitative.” It's as if they 
make a very early decision about what they're going to learn about. 
(HE researcher and TLRP Team Leader) 

My experience with miming our existing course is that students 
sometimes opt out of coming to the sessions that they don’t think 
they are going to use and it’s quite hard to persuade them that really 
this is part of their education and how are they going to be able to 
evaluate what somebody else has written if they don’t come to the 
workshops on all the different methodologies. (HE Researcher and 
research training officer) 

I think it’s partly that qualitative researchers or people who promote 
it have actually promoted it in a way that is very, very, appealing to 
students and so on. I've sat in on organised, co-ordinated, been 
involved in a lot of research methods courses and often, I know I'm 
caricaturing, its often said there are two ways of seeing the world, a 
positivist, a scientist in a white coat and you use quantitative 
methods and students sort of recoil from that or you can be an open 
and a qualitative researcher, you engage with meaning and you 



know you're a warm cuddly sort of person and students think “yeah 
yeah that’s me, I want to do that, that's me, I don't want to be a 
scientist in a white coat”. (HE researcher and TLRP Team Leader) 

The few quantitative researchers amongst the stakeholders are all too 
aware of this and how ‘being’ or being labelled a quantitative researcher 
has its problems. 

There’s actually an anti-quantitative not just kind of ignorance. In 
some quarters there are people... people like me... we get labelled 
as positivists. This is a term of abuse. That doesn’t help and it can 
be... I think there was a period in my own institution and this is 
highly selective but there was a period five or six years ago here 
with I know there were people going around criticising any kind of 
quantification. They were criticising work that I and colleagues were 
doing as being positivist in a kind of derogatory sense... Which is 
not to say that there aren’t some people who are already abusing 
quantitative techniques and that deserves criticism but you can’t be 
against quantification in general because there are few examples of 
its abuse. (HE researcher and RAE panel member); 

It's all nonsense, all of it. I'm fed up being labelled a positivist by 
people who wouldn't know positivism if they had it in their soup. 
(Chief executive of a research funding body). 

Many stakeholders felt that these mono-method identities were 
dangerous, and made the point that the choice of methods has to be 
determined after a research question has been identified, rather than 
allowing the research to be ‘method-driven’. 
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My view is that they have to decide what question they are interested 
in and then borrow the methodology from wherever in order to 
address that. (HE researcher and RAE panel member). 

And, as this stakeholder reminds us, this has to be combined with having 
an appropriate research question in order to ensure the research is 
relevant in the first place. 

There is a dual process I think of arriving at appropriate questions 
and then determining the kinds of answers and methodologies that fit 
those. (OECD researcher). 

A number of stakeholders agree that research starts with appropriate 
questions, and that this needs considerably more attention from the 
community. 

So the first thing to say is that one has to limit it to questions that are 
researchable. I think the problem with the poor quality end of your 
continuum is that very often people try to do things that are not 
possible and it is inevitable that it is going to be low quality stuff 
(HE researcher and RAE panel member). 

Behind the need to choose methods and methodologies that are 
appropriate for the questions asked, i.e. a ‘fit for purpose’ model, is the 
feeling that the stakeholders are concerned that too many researchers are 
‘mono-methodic’. In other words researchers tend to employ a single 
method or approach to their research. 
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I think on balance it’s a good thing to be problem driven. What did 
they say about single methodology people - give a child a hammer 
and everything becomes a nail. They don’t want people seeing the 
world through the methodological lenses that you put on in 
university fifteen years ago. Sounds a bit hard I know. But if 
knowledge is determined in parts by the ways in which it has been 
generated then you’ve got to have access to different ways of 
generating knowledge, at the very very least what I’m calling a 
passive competence. (Director of a research organisation). 

Perhaps research textbooks and journals tend to be dominated by 
qualitative methods, generating a greater profile for this type of research, 
and across the entire social sciences. 

But I suppose that it’s also, that if you look at what’s written on 
research methods, there’s an awful lot more written on qualitative 
methods than quantitative methods. I mean Sage publish about, you 
know, ten titles a week, as far as I can see, and that’s just one 
publisher on qualitative. They don’t publish as many on quantitative 
and they don’t publish as many on mixed methods, so I kind of think 
it’s a social science wide phenomenon... it’s certainly true of the 
other social sciences that that’s also an issue, so I don’t think it’s 
specific to education. (HE researcher) 

Are these views of the methods in general use supported by the 
researchers themselves? 
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included on the coding frame had been used in very few, if any 
submissions. Therefore it was decided to group these methods into larger 
categories to proceed with the analysis. 

• Action Research 

• Case Study 

• Comparative study 

• Computer software (for data collection and analysis), Qualitative 
data analysis software (e.g. NUD*IST, ATLAS. ti, Ethnograph), 
Quantitative data analysis software (e.g. SPSS, MlwiN, GLIM), 
Geographical information systems (e.g. Arcinfo), Programming 
languages (e.g. HTML, Hypertext, C+), IT development 

• Diaries / Autobiographical / Teaching Logs / Life history 

• Ethnomethodology 

• Experimental design. Quasi experiment 

• Group interview / Focus group / Discussion Group / Consensus 
Groups 

• Historical design / study / analysis / Archive Research 

• Intervention Research 

• Interview, Structured interview. Semi-structured interview. Un- 
structured / Informal Interview / Open Ended Interview, One-to- 
one, Telephone interview. E-mail interview, Internet relay 
interview 

• Linguistic Analysis, Semiotics, Conversational analysis 

• Literature Review / Research Review / Review / Overview / 
Literature Survey 

• Longitudinal Study 

• Non-classifiable / not really a method used 
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• Observation, Participant observation, Non-participant observation. 
Observation schedules / Systematic Observation, Unstructured 
observation 

• Philosophical study / analysis 

• Pictures / paintings / artwork. Photographs, Sound recordings. 
Video / film footage. Maps / mental maps 

• Policy study / analysis 

• Programme Evaluation 

• Psychometric / attitude / personality scales / Repertory Grid / 
Protocol Analysis / Likert Scale 

• Qualitative methods (general)/ data analysis / Unclassifiable 
Qualitative, Interpretative approach. Grounded theory approach, 
Interactionism, Phenomenology, Vignette 

• Quantitative methods (general) / data analysis / Unclassifiable 
Quantitative / Statistical analysis. Means, standard deviations. 
Frequencies, Graphs and charts. Cross-tabulations, Handling 
missing values. Corrective weightings. Probability, Set theory. 
Indices of inequality. Political arithmetic. Transforming data 
distributions. Correlation (bivariate). Regression (multivariate). 
Comparing means (e.g. t-tests, ANOVA), Comparing frequencies 
(e.g. chi-squared, Mann- Whitney), Principal components / Factor 
analysis. Classification / cluster analysis. Multi-level modelling. 
Log-linear modelling. Time-series analyses. Spatial analysis (e.g. 
nearest neighbour index) 

• Secondary numeric data sources. Statistics of Education UK 
(DIES) secondary data. School examination results / Test Results, 
UK Census of the population. National child development survey. 
Youth Cohort Study of England and Wales, British Household 




29 



28 



Panel Survey, Labour Force survey, ESRC Data Archives, 
International datasets 

• Survey, One-off survey. Repeated survey (same people). Repeated 
survey (different people). Self completion survey, Face-to-face 
survey, Telephone survey. E-mail survey, Internet survey 

• Systematic Review, Meta-analysis 

• Textual analysis / Documentary Analysis, Textual data sources 
(e.g. diaries, letters, biographies). Written work / Exam Scripts / 
Work Book, Content analysis. Discourse analysis, Narrative 
analysis. Hermeneutics, web-based textual analysis 

• Think piece / conceptual piece / theoretical development or 
analysis / curriculum commentary/analysis / Critical Analysis / 
Evaluative / Methods Research/development / Synthesis / 
Discussion, Reflective analysis 

• Written tests. Behaviour performance test / Performance Test, 
Physical / chemical tests 

Selected findings 

Table 1 shows a summary of the methods reported as having been used 
by our sample of UK education researchers. Six of the top seven are 
largely ‘qualitative’ in nature (while the other, sampling, is generic), and 
this appears to reinforce the views above. However, three of the next four 
methods are largely ‘quantitative’, and it should be noted that the ranked 
differences in frequency are not large. Around three quarters of the 
community have conducted a survey, and 65% report having used a 
secondary numeric source such as the Statistics of Examinations in 
Schools. There is, apparently, a considerable capacity to conduct 
quantitative work already. 
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Unsurprisingly, the reported use of all methods rises with the age and 
experience of the respondents. This means, of course, that in a snapshot 
study such as this we have no way of knowing whether the situation is 
improving. But it is certainly true that reported expertise is less among 
the younger researchers than the older ones, despite the reservations of 
our stakeholders (see above). Apart from this, there does not seem to be a 
large change in method approaches over time (younger cohorts are no 
more or less quantitative, for example). Men are somewhat more likely to 
report some quantitative techniques, including multivariate analysis, 
experiments, and formal tests. Women are correspondingly more likely to 
have used focus groups, diaries, conversational analysis, and 
ethnomethodology. 
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Table 1 - Frequency of used methods 



METHOD 


Number 


Percenta 

ge 


Female 


Male 


Interview 


480 


92 


93 


91 


Literature review 


471 


90 


90 


91 


Case study 


421 


81 


81 


80 


Sampling 


416 


80 


78 


83 


Observation 


415 


80 


80 


80 


Textual analysis 


409 


79 


80 


78 


Qualitative general 


400 


77 


78 


75 


Quantitative general 


393 


76 


73 


80 


Survey 


391 


75 


74 


77 


Triangulation 


370 


71 


71 


71 


Secondary numeric sources 


340 


65 


67 


64 


Group interview 


339 


65 


70 


58 


Software for collection/analysis 


325 


63 


64 


61 


Pictures/sound 


298 


57 


58 


56 


Evaluation 


289 


56 


53 


60 


Action research 


269 


51 


51 


51 


Systematic review (meta-analysis) 


254 


49 


48 


50 


Diaries 


239 


46 


48 ^ 


43 


Experiment 


212 


41 


36 


48 


Longitudinal study 


199 


39 


37 


41 


Linguistic/conversational analysis 


195 


38 


40 


34 


Formal tests 


194 


37 


35 


41 


Scales/psychometry 


184 


35 


36 


35 


Ethnomethodology 


142 


27 


30 


23 


Visual/sound sources 


91 


17 


17 


17 


Historical/archive 


69 


13 


11 


17 



N=514 



Again, Table 2 provides some confirmation for the views of the 
stakeholders. General quantitative methods are the most frequently 
reported need for training in our sample. But, again, this should not be 
over-emphasised because much of the other frequently requested training 
is in qualitative techniques. 



Demand for training in all methods declines with mean age of the 
respondent. Older researchers are, presumably, happy with their range of 
skills and less likely to participate in research capacity-building activities. 
In general, women requested somewhat more training than men and, 
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despite the above findings about previous methods usage, their requests 
are more commonly for qualitative methods. 



Table 2 - Frequency of training requests 



METHOD 


Number 


Percenta 

ge 


Female 


Male 


Quantitative general 


354 


69 


70 


67 


Software for collection/analysis 


338 


66 


70 


60 


Textual analysis 


296 


58 


56 


60 


Systematic review (meta-analysis) 


292 


57 


57 


56 


Other numeric sources 


276 


54 


53 


55 


Qualitative general 


273 


53 


57 


48 


Interview 


271 


53 


55 


49 


Sampling 


259 


50 


54 


45 


Pictures/sound 


253 


49 


54 


42 


Triangulation 


244 


48 


52 


41 


Survey 


233 


45 


48 


42 


Linguistic/conversational analysis 


231 


45 


47 


42 


Longitudinal study 


193 


38 


37 


38 


Observation 


186 


36 


36 


37 


Experiment 


185 


36 


36 


36 


Formal tests 


168 


33 


35 


30 


Group interview 


167 


33 


33 


32 


Ethnomethodology 


166 


32 


34 


30 


Evaluation 


166 ^ 


32 


37 


26 


Action research 


162 


32 


34 


29 


Historical/archive 


160 


31 


35 


26 


Diaries 


157 


31 


31 


30 


Scales/psychometry 


152 


30 


32 


26 


Visual/sound sources 


149 


29 


27 


33 


Literature review 


140 


27 


30 


23 


Case study 


117 


23 


27 


17 



N=514 



One limitation of this form of reporting is that we have no idea how often 
each respondent uses each technique, nor at what level of quality. To 
extend our search, we move now to the returns for the 2001 Research 
Assessment Exercise. 
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What do publications tell us about current capacity? 



Methods and the RAE 

Having experienced difficulties in assessing the quality of research 
returns in education for both the 1992 and 1996 RAE, a task force was 
established by the education panel members to suggest improvements to 
facilitate judgements for the forthcoming 2001 RAE. The problems 
stemmed largely from the scale of the enterprise. Education is routinely 
one of the three largest submissions to each RAE. A letter submitted to 
the task force by Bassey and Pollard (see Appendix A) contained 
suggestions for adding three additional fields of required information. 
The new areas included information about: field(s) of enquiry, prime 
audience and educational significance, theoretical and methodological 
description. These were intended to aid judgements on the scope and 
value of each submission by the panel. The guidelines by HEFCE (1999a) 
suggest that the Education panel should examine in detail 10% of the 
output, and at least 5% of each submission, making use of the 
theory /method field as ‘A succinct description of the theoretical and 
methodological approach’ (p.304). Research is defined here as an 
‘original investigation undertaken in order to gain knowledge and 
understanding... It also excludes the development of teaching materials 
that do not embody original research’ (HEFCE 1999b, p.4). 

This paper uses the information now publicly available in these additional 
fields to give a general overview of the nature of research being 
undertaken in departments of education throughout the UK. The new 
‘theory/method’ field, although limited to 100 characters, has provided a 
valuable ‘snapshot’ into education research, allowing analysis to be 




34 



33 



carried out on a much larger scale than has previously been possible 
(around 8,700 individual returns). 

Information submitted from each institution was downloaded from HERO 
njRL:<http://www.hero.ac.uk/rae/ >). and strict guidelines were followed 
by the two members of research staff involved in the coding process. The 
coding frame used to record the different methods employed in the 
research returns was based on the ESRC-fimded Research Capacity 
Building Network consultation instrument (see above), although minor 
changes were made during the coding process in light of anomalies. Each 
submission was coded for type (book, article etc.), journal title (where 
appropriate), and each method specified in the field. Of course, this 
coding process involved compromise and overcoming some problems. 
Probably the first finding from this work is that greater guidance for 
consistency is required in the completion of this field if it is to be used 
again. We need much greater consensus in the use of method terms (see 
Sheffield and Saunders 2002), probably including the creation of standard 
thesauri for searching and indexing of methods, as we already have for 
substantive areas. 

The theory /method field is a limited 100 characters, not allowing 
complete description of the complex methods utilised in all forms of 
research. There were also inaccuracies in recording, some confusion over 
what should be included, and a lack of time and effort by some 
institutions to complete this new field. McNay (2003) states that the 
additional fields for the education UoA for the 2001 RAE should 
facilitate in setting standards of research quality, which is the purpose for 
which they were intended. However he acknowledges that the new fields 
‘were not well used in their first time of operation. This was a positive 
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innovation, but may need a second round of usage to get the best from it’ 
(McNay, 2003, p.52). 

Throughout our coding process it became apparent that patterns were 
emerging for different institutions and different authors in the way the 
new fields had been completed. Consequently it was considered 
important to discover more about the RAE input procedure as the method 
for recording this information by each institution directly determines the 
results of any analysis undertaken using this field. Following discussions 
with a number of staff at Cardiff University School of Social Sciences 
involved in the 2001 RAE education submissions process, it became clear 
that there had been a level of confusion surrounding the new fields of 
information. In Cardiff the RAE is a centralised process both at an 
institutional and departmental level. Therefore, each submission was 
checked, and on many occasions amended. A number of staff members 
had clearly not read the guidelines or had not treated the new fields with 
the level of attention required. Overall, the new fields were found 
difficult to complete, and one can only imagine the different ways in 
which the field could have been interpreted in Universities with a less 
centralised process. 

Therefore, we start our analysis in the knowledge that these fields are 
limited in use and imperfectly completed. However, such a situation is 
not uncommon. In 1992 an audit of RAE submissions was undertaken to 
verify the accuracy of the RAE returns. 10 per cent of submissions were 
scrutinised and it was found that ‘whilst the audit has not identified any 
cases of excessive and unacceptable opportunism, it has identified a 
number of misinterpretations of the data requested and data entry errors’ 
(HEFCE, 1992). An audit was also undertaken in 1996, and ‘checks 
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revealed that the submissions contained a number of minor inaccuracies, 
and a smaller number of quite significant errors’ (HEFCE, 1996 p. 4). 
During our own coding process of the 2001 education RAE returns, a 
number of errors were found in the information submitted. These were 
primarily in the information submitted for journal entries (mismatching 
ISSNs, misnamed journals etc.). However, since these inaccuracies were 
found merely by coding the information, we suspect that if the 
information had been checked more thoroughly with the sources then 
many more inaccuracies would have been discovered. 

A large number of submissions had clearly undertaken empirical research 
but had not included enough information about the methods used to allow 
them to be included in the coding frame. There were also a number of 
returns which gave no indication of methods, and others that displayed 
general lack of regard for the information required. We quote here a few 
examples of the ‘completed’ theory /method field that would give 
problems to any methods classification system. 

• Empirical investigation of visually impaired children’s play and 
language. 

• Analysis of course materials 

• Uses two projects to illuminate strengths and weaknesses of model 

• Gender studies: analysis of experiences & attainments of 
academically able boys 

• Research 

• Cognitive development 

• Empirical study of learning: comparison of approaches 
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It is important, we feel, that the problems associated with re-analysis of 
this field are discussed, so that the findings are viewed with appropriate 
caveats. However, we also wish to stress that (unlike the richer data used 
above) this analysis enables us to make a comparison of around 8,700 
research pieces - the largest such overview of methods in education 
research. 



Selected findings 

Table 3 summarises the type of each of the 8,700 publications submitted 
to the education panel. While each publication was submitted by an 
individual, they are grouped here in terms of the RAE outcome for the 
centre by which they were submitted (graded 1 to 5*). Only books, edited 
books, journal articles, conference publications and reports to external 
bodies feature heavily, and they appear in roughly the same proportions 
for all RAE outcomes (only one submission was rated 1). Therefore, there 
is not a mode of publication that is disproportionately high or low ranked. 
In fact for submissions rated 3a and above, the relative frequencies for 
each type are surprisingly similar. For the only submission rated 1, 
conference papers tend to appear in place of authored books and chapters 
(but not journal articles). For the eight submissions rated 2, however, 
chapters in books tend to appear in place of journal articles (perhaps an 
indication that this medium is less highly considered or less involved in 
peer-review). For the two submissions rated 5* there is a higher 
proportion of reports to external bodies. This may not be important, or it 
could be seen as a presage of the kind of high impact and ‘third mission’ 
dissemination needed for success in future RAEs. 
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Table 3- Percentage of each publication type by RAE outcome 





1 


2 


3a 


3b 


4 


5 


5* 


All 


Authored book 


7 


12 


10 


12 


10 


15 


12 


12 


Edited book 


2 


2 


3 


2 


2 


2 


2 


2 


Chapter in book 


9 


32 


22 


21 


23 


25 


24 


23 


Journal article 


57 


43 


58 


52 


58 


53 


52 


55 


Conference 


22 


6 


4 


5 


3 


1 


2 


3 


paper 


















Software 


0 


1 


0 


0 


0 


0 


0 


0 


Report ext. body 


2 


3 


3 


4 


4 


3 


6 


4 


Internet 


0 


1 


0 


1 


0 


0 


1 


0 


publication 


















Internet 


0 


0 


0 


0 


0 


0 


0 


0 


subscription 


















Exhibition 


0 


0 


0 


0 


0 


0 


0 


0 


Scholarly edition 


2 


0 


0 


0 


0 


0 


0 


0 


Other 


0 


0 


0 


0 


0 


0 


0 


1 


TOTAL 


100 


100 


100 


100 


100 


100 


100 


100 



Note: percentages have been rounded. 



There were over 1,000 journals represented in all submissions to 
education, a staggering number suggesting considerable diversity in 
approach and audience. In fact, this works out at around one journal for 
every two of the 2,045 UK education researchers submitted to the 2001 
RAE. Table 4 shows the frequency of the 50 most frequently submitted 
journals, their relative rank for submissions rated 5/5* and 1-4, and their 
Social Science Citations Index impact score (where the journal is part of 
the index - itself an indicator of impact in some ways). Table 4 shows 
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several interesting patterns (and the following section considers the extent 
to which articles in a subset of these journals can be labelled quantitatove 
or qualitative). The majority of these highly-placed journals in education 
are not in the SSCI, and the relative frequency of an impact score declines 
in the table, and more so for the journals outside the top 50 (and not 
featured here). In general, higher impact journals (and journals with an 
impact score) are somewhat more closely associated with submissions 
ranked 5/5*. The highest ‘impact’ journal here. Comparative Education, 
only appears in the top 50 for 5/5*, for example. So do the British Journal 
of Educational Psychology, the International Journal of Educational 
Development, and the Journal of Philosophy of Education. Even so, there 
are ‘impact’ journals that show the opposite. Computers in Education has 
a reasonable impact score, and is the fourth most frequent journal for 
submissions rated 1-4, but does not appear in the top 50 for 5/5*. There 
are, of course, problems with the SSCI impact scores. They 
disproportionately favour US-relevant publications, and being part of the 
Index is not based solely on impact, but also on balance between titles 
and fields. 

Table 4 also shows that despite the large overall number of outlets for 
publication, the RAE-relevant publications for education tend to cluster in 
a relatively small number of titles (further details in Appendix B). The 
majority of the 1,006 different journals submitted to UoA68 appear only 
once (such as Information Technology in Nursing, or the Journal of 
Property Research). It should therefore be emphasised that, for the 
purposes of the following discussion, all of the journals mentioned are 
among the most highly-submitted for education. Dividing the 
submissions into 1-4 and 5/5* also divides the titles into three types. The 
first type is represented by the British Educational Research Journal, the 



International Journal of Science Education, the Journal of Education 
Policy, Research Papers in Education, and so on. Articles in these 
journals were commonly submitted by authors in both outcome groups 
equally. As we would expect for ‘popular’ journals the top 50 tends 
anyway to emphasise the generic titles which publish empirical work 
almost irrespective of discipline and subject matter. The second-ranked 
journal, the International Journal of Science Education, is an interesting 
divergence from this ‘generic’ trend. 

The second group of journals in Table 4 are those disproportionately 
presented by researchers in 5/5* submissions. These include the British 
Journal of Educational Psychology (already mentioned). Assessment in 
Education, Comparative Education, the British Journal of Guidance and 
Counselling, the International Journal of Educational Development, and 
so on. It is interesting to consider whether these are specifically ‘5’ 
journals because of their impact, quality, and peer-review, or whether 
they represent areas or types of research especially prevalent in some 5- 
rated departments. 

The third group of journals is disproportionately represented in 1-4 rated 
submissions. These include Computers in Education (as above). 
Educational Management and Administration, Educational Research, 
Assessment and Evaluation in Higher Education, the Journal of 
Vocational Education and Training, and so on. It is interesting that both 
of the leading UK education management journals were more often 
presented by 1-4 departments, and might be due to the focus of this work 
being conducted outside what are currently 5/5* departments. Similar 
points could be made about research into post-compulsory education 
(including the higher and vocational sectors). 
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Table 4 - Frequency of articles by journal and impact 



Rank 


Journal 


Freq 

uenc 




1-4 

rank 


5/5* 

rank 


Impa 

ct 


1 


British Educational Research Journal 


100 


1 


1 




2 


International Journal of Science Education 


80 


2 


2= 


0.48 


3 


Oxford Review of Education 


64 


13= 


2= 


0.30 


4 


Journal of Education Policy 


58 


7= 


5= 


0.62 


5= 


British Journal of In-Service Education 


57 


3 


16= 


- 


5= 


Research Papers in Education 


57 


9= 


5= 


- 


7 


British Journal of Educational Studies 


56 


5 


13= 


0.78 


8 


British Journal of Sociology of Education 


55 


12 


4 


0.71 


9 


The Curriculum Journal 


49 


18= 


7 


- 


10 


Educational Studies 


48 


6 


26= 


0.32 


11 


Teacher Development 


46 


15= 


11 


- 


12 


Computers in Education 


45 


4 


- 


0.57 


13= 


Cambridge Journal of Education 


44 


25= 


9 


- 


13= 


Educational Review 


44 


9= 


23= 


0.25 


15 


Educational Management and Administration 


40 


7= 


- 


- 


16 


School Leadership and Management 


39 


13= 


26= 


- 


17 


Educational Research 


37 


11 


- 


0.36 


18= 


International Journal of Inclusive Education 


36 


29= 


16= 


- 


18= 


Studies in Higher Education 


36 


20= 


26= 


0.52 


20= 


Compare 


34 


34= 


18= 


- 


20= 


Gender and Education 


34 


25= 


26= 


- 


20= 


Journal of Further and Higher Education 


34 


20= 


39= 


- 


20= 


Teaching in Higher Education 


34 


18= 


44= 


- 


24= 


British Journal of Educational Psychology 


33 


- 


10 


0.75 


24= 


Journal of Education for Teaching 


33 


45= 


13= 


- 


26 


Association for Learning Technology Journal 


32 


23= 


44= 




27= 


Assessment and Evaluation in Higher Educ. 


31 


23= 






27= 


History of Education 


31 


28= 


39= 




27= 


Journal of Vocational Education and Training 


31 


15= 






27= 


Research in Education 


31 


20= 






31= 


Assessment in Education 


30 




8 




31= 


Educational Psychology 


30 


38= 


23= 




31= 


International Journal of Lifelong Education 


30 


15= 






34 


Medical Education 


29 


29= 


44= 




35 


British Journal of Special Education 


28 


41= 


26= 




36= 


Comparative Education 


27 




23= 


1.02 


36= 


Journal of Computer Assisted Learning 


27 


36= 


44= 


0.23 


38= 


British Journal of Guidance and Counselling 


26 




29= 


0.47 


38= 


Innovations in Education and Training Inti. 


26 


25= 




0.19 


38= 


International Journal of Educational Devpt. 


26 




13= 


0.33 


38= 


Journal of IT for Teacher Education 


26 


45= 


33= 




38= 


Journal of Research in Reading 


26 




20= 
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38= 


Research in Science and Technology Educ. 


26 




18= 




44 


Higher Education Quarterly 


25 


36= 






45= 


Changing English 


24 




12 




45= 


International Journal of Early Years Educ. 


24 


45= 


44= 




45= 


International Studies in Sociology of Educ. 


24 




20= 




45= 


Journal of Philosophy of Education 


24 




26= 


0.42 


45= 


Language Learning Journal 


24 




39= 




45= 


Support for Learning 


24 









For all submissions, whether journal articles or not, Table 5 shows the 
frequency of the collapsed categories for their methods descriptions. A 
very large number of submissions were based on think pieces (18% 
overall), or literature reviews (10%), or were non-empirical or otherwise 
unclassiflable by method (4%, see above). Thus, over 32% of the total 
submission to education was not clearly empirical (according to the 
theory/method field), and this has not, apparently, hindered success since 
this 32% applies to 5/5* as much as 1-4. The only hint of a difference 
here lies in the unclassifiable returns which were more prevalent in 1-4 
returns. Many of the examples given above that caused us problems in the 
coding process were in 1-4 returns. Perhaps these institutions had a less 
centralised checking system for RAE returns. 

Of the remaining 68% of submissions with a clearly flagged empirical 
component, the most common single methods were surveys and case 
studies (8% each) - one apparently quantitative and one perhaps more 
qualitative. This balance might surprise our stakeholders (reported above) 
who lamented the lack of quantitative work in education. If we add the 
frequencies for surveys, quantitative (unspecified in the field), 
experiments, scales, formal tests, and secondary numeric data, we find 
that over 14% of the returns were clearly or largely quantitative. If we 
add together case studies, qualitative (unspecified), interviews, textual 
analysis, ethnomethodology, linguistic analysis, group interviews, and 
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diaries, we find that 28% of the returns were clearly or largely qualitative. 
The balance is therefore 2:1, a large difference but perhaps less than 
previous commentaries have suggested, and one that confirms the 
findings from our survey (see above). The prevalence of all methods is 
remarkably similar across the two groups defined by RAE outcomes. 

Naturally, many of the authors cited two or more distinct methods (and 
we have run the analysis underlying Table 5 using all methods cited, 
leading to higher frequencies but without changing the overall pattern). 
The most common combination is the use of interviews and surveys 
together. In addition, some of the method descriptions used by authors do 
not allow us to categorise them definitively as either qualitative or 
quantitative. Examples include action research, interventions, case 
studies, and comparative work. We might be disposed to believe that 
these are all largely qualitative in practice, but the theory/method field 
does not specify this. This is one of the reasons that we look, in the next 
section, at the methods used in a selection of journals. 

Two notable features of Table 5 are the paucity of experiments and 
secondary numeric data (both less than 1% of the total) among the pieces 
submitted to the RAE as described by the theory/method field. This is 
notable in light of the reports of researchers in the previous section that 
41% have conducted an experiment, and 65% have used secondary 
numeric data. It is not clear why these methods should be selectively 
ignored by authors when completing the theory/method field, and we may 
conclude that while many people know how to work with both 
approaches, these are not actually common (see also next section). 
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Table 5 - Frequency of reported first or main method 



METHOD 


Number 


Percentag 

e 


Percentag 
e RAE 1- 
4 


Percentag 
e RAE 5- 
5* 


Think piece 


1533 


18 


17 


19 


Literature review 


828 


10 


10 


10 


Survey 


697 


8 


8 


8 


Case study 


674 


8 


9 


5 


Qualitative unspecified 


494 


6 


6 


5 


Comparative 


479 


6 


5 


6 


Policy study/analysis 


465 


5 


5 


6 


Interview 


407 


5 


5 


5 


Textual analysis 


392 


5 


4 


5 


Not classifiable/no method 


364 


4 


5 


3 


Historical/archive 


344 


4 


4 


5 


Quantitative unspecified 


271 


3 


3 


4 


Ethnomethodology 


268 


3 


3 


4 


Action research 


233 


3 


3 


2 


Philosophy 


191 


2 


2 


3 


Observation 


190 


2 


2 


2 


Programme Evaluation 


131 


2 


1 


2 


Longitudinal study 


121 


1 ^ 


1 


2 


Linguistic/conversational analysis 


102 


1 


1 


2 


Experiment 


94 


1 


1 


1 


Scales/psychometry 


80 


1 


1 


1 


Software for collection/analysis 


65 


1 


1 


0 


Group interview 


51 


1 


1 


0 


Formal tests 


46 


1 


1 


0 


Diaries 


37 


0 


0 


1 


Systematic review (meta-analysis) 


31 


0 


1 


0 


Pictures/sound 


29 


0 


0 


0 


Intervention 


25 


0 


0 


0 


Secondary numeric data 


15 


0 


0 


0 


Total 


8691 


100 


100 


100 



Because of the nature of the RAE we are able to use this data to make 
some tentative judgements about quality. There is clearly some good 
research in education using both quantitative and qualitative approaches. 
However, we have still not satisfactorily solved the issue of judging 
prevalence. The theory/method field is limited, and relies on 
unstandardised self-reporting. So we move to a brief assessment of what 
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is being published, both to triangulate, to some extent, our analyis here 
and to consider further the differences between journals illustrated above. 



What is being published? 

For the purposes of this brief analysis we selected three journals, and 
collated the methods used in all papers for the calendar year 2002 (the 
most recent complete year at time of writing). We chose for this purpose 
BERJ, the most commonly submitted to the RAE, the British Journal of 
Educational Psychology, a disproportionately 5/5* publication, and 
Educational Management and Administration, the most clearly 1-4 
publication (see above). 

BERJ 

As noted above the journal BERJ was the most widely submitted outlet 
for RAE papers. In 2002, the most recent complete year, BERJ contained 
42 articles in six issues and these showed several differences to the RAE 
returns more generally. The category ‘unclassifiable’ from the RAE 
analysis on the basis of the method field (3%) obviously does not exist 
for papers in BERJ, nor are any of the pieces stand-alone literature 
reviews (as befits a research-based journal). This means that the other 
categories should be larger. Only five articles (12%) were apparently 
non-empirical ‘think pieces’ compared to 18% in the RAE. The most 
common category is interviews, occurring in a total of 18 papers (43%), 
much higher proportionately than in the RAE (17% interviews, case 
studies, ethnomethodology, and conversational analysis). The BERJ 
figure includes 1 1 pieces simply with interviews, one piece simply with 
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interviews but related to a companion piece involving regression analysis, 
two combining interviews and surveys, and one each with discourse 
analysis, case study (with interviews but also documents and 
observation), interview with observation, and interview with textual 
analysis. Four BERJ papers used texts (usually pre-existing), one was 
based on ‘qualitative’ observation, and one on autobiography. 

Of the remainder, 13 (or 31%) were clearly ‘quantitative’ in nature, not 
counting a factor analysis that also appeared in one of the papers above. 
Another paper used factor analysis with psychometric scales, three used 
multi-level modelling in a school-effectiveness vein, and three used 
regression with secondary data (but not MLM). Two papers reported a 
questionnaire survey with comparison of means, one a survey with 
regression, and one a survey with a school-based test. There was a further 
paper using secondary data for simple numeric analysis, and one 
conducting observation leading to numeric outcomes. Thus, numbers- 
based papers were a significant component of BERJ output, and exhibited 
considerable variation in technique (again contrary to the commentary 
with which this paper started). 

BJEP 

The British Journal of Educational Psychology published 28 articles in 
four issues during 2002 (with no special issues). Of these, all but one 
(96%) were clearly or largely quantitative — the odd one being a literature 
review. There were five experiments or quasi-experiments (18%), a much 
higher figure than all other sources (see above). Seven of the papers were 
based on data collected via a questionnaire (25%) and then analysed via 
factor analysis (3), Rasch modelling, comparison of means (2), multi- 
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level modelling, regression and cluster analysis. Five were based on 
formal tests, variously with a questionnaire, comparison of means (3), 
and factor analysis. One further paper used factor analysis on pre-existing 
data, one used growth curves, and one each correlation, Rasch modelling, 
and longitudinal assessment data. The prevalence, high level and variety 
of quantitative techniques may be a function of the disciplinary nature of 
this journal, but it is also notable that this outlet clearly distinguishes 
between high and low-medium RAE outcomes. 

EMA 

Educational Management and Administration published 24 articles in 
four issues during 2002 (including a special issue on management 
theory). Of these 7 (29%) were think pieces with no discernible empirical 
content, and a further three (9%) were literature reviews. Six were based 
on interviews (18%), including one in conjunction with textual analysis, 
and another with secondary numeric data. Three papers described 
‘qualitative’ case studies, one was comparative, one a policy analysis, and 
three used questionnaire with analysis of frequencies and means. 
Therefore EMA, which was disproportionately submitted by departments 
gaining RAE 1-4, contained less empirical work overall than the other 
journals considered, and also considerably less quantitative work. Perhaps 
it is fields and journals such as these that lead commentators to the idea 
that there is a marked lack of quantitative skills. 



Discussion 



Overall, across three journals in 2003, 17% of the articles were clearly or 
largely non-empirical (although this description includes literature 
reviews based on empirical evidence), 4% were empirical pieces using a 
combination of ‘qualitative’ and ‘quantitative’ methods (therefore a rather 
rare phenomenon), 34% used qualitative methods alone, and 47% used 
quantitative methods alone. While the selection of journals used here may 
indeed overemphasise quantitative approaches (although there is no 
certainty that Assessment in Education would be any different to B JEP , in 
this respect), this simple analysis of published papers in one year suggests 
that at least one element of our stakeholders view of problems in 
education research is incorrect. There is no particular shortage of 
quantitative work as evidenced by any of our indicators - the reports of 
researchers themselves, RAE returns, and journal contents. 

If there is, indeed, more quantitative work going on in education than is 
usually realised by commentators, then why is there this discrepancy? 
There are several possible explanations. First, of course, the distinction 
between quantitative and qualitative work is not a clear one, and taken 
literally almost all work involves both textual analysis and counting, so 
our classifications may be invalid. Second, our stakeholders generally 
talked about education research as an activity in higher education 
institutions in specialist education (largely teacher training) departments. 
Much research relevant to education goes on outside of these, in other 
disciplines (such as psychology), in governmental structures, and in 
outside bodies such as NERF or NIACE. Perhaps the distinction is not so 
much within education HEIs (our analysis of RAE returns shows little 
difference between high and low ranked departments in their use of 
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methods), but between HEIs (who may do less quantitative work) and 
everyone else. Our stakeholders were not, therefore, bemoaning the lack 
of quantitative work in general, but in what are mainly teacher training 
departments. 

Third, merely having work labelled ‘quantitative’ is of no consequence if 
it is of obviously poor quality, which is what several stakeholders 
complain about. Perhaps they mean that there is little ‘good’ quantitative 
work. In the same way that there are complaints about people simply 
taping a few unstructured interviews with a convenience sample, and then 
using grounded theory to extract a few quotations to make a paper, ‘by 
numbers’ as one of our informant said, there are complaints that 
quantitative work should not be simply assigning numbers to 
questionnaire responses and then pushing some buttons on SPSS. There 
has been a recent growth in demand for more field experiments in public 
sector research (e.g. Shavelson and Towne 2002), in order to find out 
what works in realistic settings. Our summary of three sources suggests 
that very little of such work is going on in education. In fact, in our 
review there were more articles decrying the academic hegemony of 
‘what works’, and bemoaning the fact that experiments were swamping 
the field than there were actual field experiments. Similar comments 
apply to the published use of secondary numeric data. Such data is being 
produced by governments, or archived by research councils, and 
advocated by stakeholders, but little use appears to be made of it. Only 15 
pieces out of 8,691 RAE returns, and two articles in our three journals 
used existing large-scale datasets for any significant reanalysis (and both 
of these were by the same author). 
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One practical conclusion of this re-analysis would be that improvement in 
the quality (and relevance) of UK education research is unlikely to come 
via a magic bullet of merely increasing the proportion of work deemed 
quantitative. Rather, we may need an improvement in all forms of 
research, a greater understanding of a wide range of approaches, clear 
research questions, fit between questions and methods (not methods 
identities), and perhaps most pressingly a greater transparency in our 
warrants. It may be the process of transforming research findings into 
useful and relevant knowledge that requires more care (Gorard 2003). All 
of these are issues also raised in our stakeholder interviews. For example, 
researchers need to take more care when suggesting how relevant their 
research is to policy-makers and/or practitioners or when warranting their 
findings. 

One of the things that I am always concerned about is the way that 
educational researchers express the claims that they make and I do 
think that with an experienced educational researcher, the way that 
they make claims for their exploratory studies is pretty dicey. (HE 
researcher and RAE panel member) 

For it is not weak research per se that does damage to the field 
(presumably this is a necessary concomitant of innovation and growth), it 
is the unwarranted claims based on weak research that do the harm. Of 
course, it is possible to argue that since our stakeholders appear to have 
been misled about the amount of quantitative work going on, then they 
may be wrong about the other things as well. In fact, one of the 
conclusions of this paper is to reinforce the irony that much of the 
criticism of the quality and relevance of education research evidence is 
itself evidence-free. 
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Appendix A 

Extract from a paper submitted to the RAE task group by Pollard and 
Bassey (URL: 

< http://www .niss.ac.uk/education/hefc/rae2001/edtaskgrp.html>) 

It is suggested that the RA2 returns in Education have three additional 
fields as descriptions for publications that could help the panel in making 
its judgements. These are: field(s) of enquiry, prime audience and 
educational significance, theoretical and methodological description. 

Theoretical and methodological description. Here ‘theoretical’ refers to 
any conceptual framework within which the research may be set. 
‘Methodological’ is taken to mean research approach and arena for data 
collection, analysis and interpretation. (One line only - not more than 64 
alphanumeric characters). 

Some illustrations show how this could illuminate the title of an article 
with value: 

Ethnographic case studies in 3 primary schools in Kent 
Random survey of 10% of secondary schools in Lancashire 
Action research on changing management in an FE college 
Slavin-type review of 33 studies of calculators in primary ed 
Post-modernist critique of UK policy on teacher education 
Master’s level text on qualitative research in education 
Geographical analysis of LMS funding documents 
We suggest that this would alleviate the problem met by the previous 
panel of knowing what a paper is about when it is not possible to 
actually read it. 
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Appendix B 



Listed below are the remaining journals (and their ranking) appearing in 
the top 50 most frequently submitted to the 2001 RAE in terms of RAE 
outcomes. The first group appear in the top 50 for submissions rated 5/5* 
but not those rated 1-4. The next group is the obverse. Where available, 
the r ank ing score is followed by the impact score. The latter reinforce the 
point made above — as we read down the list of highly-submitted journals 
the impact scores tend to decline in size and most clearly in frequency, 
and they are more common for journals more highly-submitted by 5/5* 
outcomes. 

5/5* 

International Journal of Educational Research 20= 

Children and Society 26= 

School Effectiveness and School Improvement 33=, 0.24 
Environmental Education Research 33= 

Science and Education 33=, 0.84 
Emotional and Behavioural Difficulties 33= 

Educational Studies in Maths 33= 

Reading 39= 

Educational and Child Psychology 39= 

For the Learning of Mathematics 44= 

Teaching and Teacher Education 44=, 0.4 



lA 

Journal of Design and Technological Education 29= 
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Scottish Educational Review 29= 

Mentoring and Tutoring 38= 

Research in Post-Compulsory Education 38= 
Directions - Journal of Art and Design Education 41= 
Quality Assurance in Education 41= 

Welsh Journal of Education 41= 

School Science Review 45= 

British Journal of Educational Technology 45=, 0.44 
Educational Action Research 50 
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